Papers with causal language modeling task
A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Mechanisms of the Transformer architecture for causal language modeling are not well understood. |
| Approach: | They propose a meta-learning view of the Transformer architecture when trained for a causal language modeling task by explicating an inner optimization process that may happen within the Transformer. |
| Outcome: | The proposed model is based on a self-attention mechanism and has been widely used in natural language processing, computer vision, and scientific discovery. |